Security system, security method, and non-transitory computer readable medium

ABSTRACT

A security system (10) includes an image information acquisition unit (11) that acquires input image information on an image taken of a person in a store, a tracking unit (12) that tracks an action of a hand of the person based on the input image information, and a suspicious action detection unit (13) that detects a suspicious action of the person based on the tracked action of the hand. A security system, a security method, and a security program capable of accurately detecting a suspicious action are thereby provided.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is a continuation application of U.S. patentapplication Ser. No. 16/126,818 filed on Sep. 10, 2018, which is acontinuation application of U.S. patent application Ser. No. 14/916,702filed on Mar. 4, 2016, which is a National Stage Entry of internationalapplication PCT/JP2014/004583 filed on Sep. 5, 2014, which claims thebenefit of priority from Japanese Patent Application 2013-185130 filedon Sep. 6, 2013, the disclosures of all of which are incorporated intheir entirety by reference herein.

TECHNICAL FIELD

The present invention relates to a security system, a security method,and a non-transitory computer readable medium storing a security programand, particularly, to a security system, a security method, and anon-transitory computer readable medium storing a security program usingperson images.

BACKGROUND ART

Damage caused by shoplifting by customers, misappropriation by part-timeemployees and the like are a continuous and growing concern for stores.In order to prevent such misconduct, a store staff or a store managerkeeps an eye on these customers and part-time employees, or recordsimages monitored by a general 2D camera and visually checks themafterwards.

Since it is inefficient to visually check for the occurrence ofmisconduct, the systems disclosed in Patent Literatures 1 to 5 asrelated art, for example, are under development.

CITATION LIST Patent Literature

-   PTL1: Japanese Unexamined Patent Publication No. 2011-065328-   PTL2: Japanese Unexamined Patent Publication No. 2010-094332-   PTL3: Japanese Unexamined Patent Publication No. 2009-048430-   PTL4: Japanese Unexamined Patent Publication No. 2009-009231-   PTL5: Japanese Unexamined Patent Publication No. 2008-257487

SUMMARY OF INVENTION Technical Problem

For example, according to the techniques disclosed in related art likePatent Literature 1, when the number of times a store staff's faceswings is a specified value of more and the cancelled amount of money ina cash register is a reference value or more, it is detected that asuspicious conduct (suspicious action) has occurred.

However, because the technique of the related art performs detectionbased on a swing of a face or the like, it fails to detect a suspiciousaction of a shop staff and the like in some cases. For example, althougha suspicious action is often carried out by a hand, the technique of therelated art cannot detect the behavior on the basis of a hand action.

Thus, the technique disclosed in the related art has a problem that itis difficult to accurately detect a suspicious action of a store staff,a customer and the like.

In light of the above, an exemplary object of the present invention isto provide a security system, a security method, and a non-transitorycomputer readable medium storing a security program capable ofaccurately detecting a suspicious action.

Solution to Problem

A security system according to an exemplary aspect of the presentinvention includes an image information acquisition unit that acquiresinput image information on an image taken of a person in a store, atracking unit that tracks an action of a hand of the person based on theinput image information, and a suspicious action detection unit thatdetects a suspicious action of the person based on the tracked action ofthe hand.

A security method according to an exemplary aspect of the presentinvention includes acquiring input image information on an image takenof a person in a store, tracking an action of a hand of the person basedon the input image information, and detecting a suspicious action of theperson based on the tracked action of the hand.

A non-transitory computer readable medium storing a security programaccording to an exemplary aspect of the present invention causes acomputer to perform a security process including acquiring input imageinformation on an image taken of a person in a store, tracking an actionof a hand of the person based on the input image information, anddetecting a suspicious action of the person based on the tracked actionof the hand.

Advantageous Effects of Invention

According to the exemplary aspects of the present invention, it ispossible to provide a security system, a security method, and anon-transitory computer readable medium storing a security programcapable of accurately detecting a suspicious action.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing main elements of a security systemaccording to an exemplary embodiment.

FIG. 2 is a block diagram showing the configuration of a security systemaccording to a first exemplary embodiment;

FIG. 3A is a diagram showing a configuration example of a 3D cameraaccording to the first exemplary embodiment;

FIG. 3B is a diagram showing a configuration example of a 3D cameraaccording to the first exemplary embodiment;

FIG. 4 is a block diagram showing a configuration of a distance imageanalysis unit according to the first exemplary embodiment;

FIG. 5 is a flowchart showing the operation of the security systemaccording to the first exemplary embodiment;

FIG. 6 is a flowchart showing the operation of a distance image analysisprocess according to the first exemplary embodiment;

FIG. 7 is a flowchart showing the operation of an alert informationgeneration process according to the first exemplary embodiment;

FIG. 8 is an explanatory diagram illustrating the operation of the alertinformation generation process according to the first exemplaryembodiment;

FIG. 9 is a block diagram showing the configuration of a security systemaccording to a second exemplary embodiment; and

FIG. 10 is a block diagram showing a configuration of a distance imageanalysis unit according to the second exemplary embodiment.

DESCRIPTION OF EMBODIMENTS Overview of Exemplary Embodiment

Prior to describing exemplary embodiments, the overview of thecharacteristics of the exemplary embodiments is described hereinbelow.FIG. 1 shows main elements of a security system according to anexemplary embodiment.

As shown in FIG. 1, a security system 10 according to this exemplaryembodiment includes an image information acquisition unit 11, a trackingunit 12, and a suspicious action detection unit 13. The imageinformation acquisition unit 11 acquires input image information, whichis an image taken of a person in a store. The tracking unit 12 tracks ahand action of a person based on the acquired input image information.The suspicious action detection unit 13 detects a suspicious action of aperson based on the tracked hand action.

As described above, in the exemplary embodiment, a hand action of aperson in a store is tracked, and a suspicious action is detected basedon the tracking result. For example, by tracking a hand action of acustomer or a store staff in front of a product shelf in a store, it ispossible to accurately detect a suspicious action that can lead toshoplifting or misappropriation.

First Exemplary Embodiment

A first exemplary embodiment is described hereinafter with reference tothe drawings. FIG. 2 is a block diagram showing the configuration of asecurity system according to this exemplary embodiment. This securitysystem is a system that detects a suspicious action of a customer or astore staff in a store or the like and outputs (displays) an alert(alarm) and the like. Note that customer includes all persons who cometo (enter) a store, and the store staff includes all persons who work ina store.

As shown in FIG. 2, a security system 1 according to this exemplaryembodiment includes a security device 100, a 3D camera 210, a facialrecognition camera 220, an in-store camera 230, and an alert device 240.For example, while the respective components of the security system 1are placed in the same store, the security device 100 or the alertdevice 240 may be placed outside the store. Although it is assumed inthe following description that the respective components of the securitysystem 1 are separate devices, the respective components may be one orany number of devices.

The 3D (three-dimensional) camera 210 is an imaging device (distanceimage sensor) that takes an image of and measures a target and generatesa distance image (distance image information). The distance image (rangeimage) contains image information which is an image of a target takenand distance information which is a distance to a target measured. Forexample, the 3D camera 210 is Microsoft Kinect (registered trademark) ora stereo camera. By using the 3D camera, it is possible to recognize(track) a target (a customer's action or the like) including thedistance information, and it is thus possible to perform highly accuraterecognition.

As shown in FIGS. 3A and 3B, in order to detect a suspicious action by ahand of a customer or a store staff, the 3D camera 210 takes an image ofa customer or a store staff at a specified position in a store in thisexemplary embodiment. In the example of FIG. 3A, the 3D camera 210 takesan image of a product shelf (product display shelf) 300 on which aproduct 301 is placed (displayed), and particularly takes an image of acustomer 400 who is about to touch the product 301 in front of theproduct shelf 300. The 3D camera 210 takes an image of a productplacement area of the product shelf 300 and an area where a customerpicks up/looks at a product in front of the product shelf 300, which isa presentation area where a product is presented to a customer in theproduct shelf 300. The 3D camera 210 is placed at a position whereimages of the product shelf 300 and the customer 400 in front of (in thevicinity of) the product shelf 300 can be taken, which is, for example,above (the ceiling etc.) or in front of (a wall etc.) of the productshelf 300, or in the product shelf 300.

In the example of FIG. 3B, the 3D camera 210 takes an image of acheckout stand 310 where a cash register 311 is placed, and particularlytakes an image of a store staff 410 who is standing in front of thecheckout stand 310 and about to sell the product 301 to the customer 400or the store staff 410 who is about to touch money 302. The 3D camera210 is placed at a position where images of the checkout stand 310 andthe store staff 410 in front of (in the vicinity of) the checkout stand310 can be taken, which is, for example, above (the ceiling etc.) or infront of (a wall etc.) of the checkout stand 310, or on the checkoutstand 310 (cash register 311).

Note that, although an example in which the 3D camera 210 is used as adevice that takes images of the product shelf 300 and the checkout stand310 is described below, it is not limited to the 3D camera but may be ageneral camera (2D camera) that outputs only images taken. In this case,tracking is performed using the image information only.

Each of the facial recognition camera 220 and the in-store camera 230 isan imaging device (2D camera) that takes and generates an image of atarget. The facial recognition camera 220 is placed at the entrance of astore or the like, takes an image of a face of a customer who comes tothe store and generates a facial image to recognize the customer's face.The in-store camera 230 is placed at a plurality of positions in astore, takes an image of each section in the store and generates anin-store image to detect the congestion of customers in the store. Notethat each of the facial recognition camera 220 and the in-store camera230 may be a 3D camera. By using a 3D camera, it is possible toaccurately recognize the customer's face or the congestion in a store.

The alert device 240 is a device that notifies (outputs) alertinformation to a surveillant such as a store manager, a business manageror a security guard and performs recording. The way to transmit (output)alert information to a surveillant is not limited, and it may be adisplay of letters and images on a display device, audio output througha speaker or the like. The alert device 240 is placed at a positionwhere a surveillant can view (hear) the alert information. The alertdevice 240 may be an employee terminal in a shelf, a cash register, aguard's room or a store, or it may be a surveillance device connected tothe outside of a store via a network. For example, the alert device 240is a computer including a display device and a storage device, such as apersonal computer or a server computer.

As shown in FIG. 2, the security device 100 includes a distance imageanalysis unit 110, a person recognition unit 120, an in-store situationanalysis unit 130, an alert information generation unit 140, asuspicious action information DB (database) 150, a 3D video informationrecording unit 160, and a suspicious person information DB 170. Notethat, although those blocks are described as the functions of thesecurity device 100 in this example, another configuration may be usedas long as the operation according to this exemplary embodiment, whichis described later, can be achieved.

Each element in the security device 100 may be formed by hardware orsoftware or both of them, and may be formed by one hardware or softwareor a plurality of hardware or software. For example, the productinformation DB 150 and the customer information DB 160 may be storagedevices connected to an external network (cloud). Each function (eachprocessing) of the security device 100 may be implemented by a computerincluding CPU, memory and the like. For example, a security program forperforming a security method (security process) according to theexemplary embodiments may be stored in a storage device, and eachfunction may be implemented by executing the security program stored inthe storage device on the CPU.

This security program can be stored and provided to the computer usingany type of non-transitory computer readable medium. The non-transitorycomputer readable medium includes any type of tangible storage medium.Examples of the non-transitory computer readable medium include magneticstorage media (such as floppy disks, magnetic tapes, hard disk drives,etc.), optical magnetic storage media (e.g. magneto-optical disks),CD-ROM (Read Only Memory), CD-R, CD-R/W, and semiconductor memories(such as mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flashROM, RAM (Random Access Memory), etc.). The program may be provided to acomputer using any type of transitory computer readable medium. Examplesof the transitory computer readable medium include electric signals,optical signals, and electromagnetic waves. The transitory computerreadable medium can provide the program to a computer via a wiredcommunication line such as an electric wire or optical fiber or awireless communication line.

The distance image analysis unit 110 acquires a distance image generatedby the 3D camera 210, tracks a detection target based on the acquireddistance image, and recognizes its action. In this exemplary embodiment,the distance image analysis unit 110 mainly tracks and recognizes a handaction of a customer or a store staff. The distance image analysis unit110 refers to the suspicious action information DB 150 to recognize asuspicious action of a customer or a store staff contained in thedistance image. Further, the distance image analysis unit 110 performsdetection necessary for recognition of a suspicious action,determination of a suspicion level and the like. For example, thedistance image analysis unit 110 detects a time period during which asuspicious action is carried out, the quantity of target products, theamount of money, the scale of a target act (the size of damage etc.) andthe like as well. Further, the distance image analysis unit 110 recordsthe distance image acquired from the 3D camera 210 as a 3D video in the3D video information recording unit 160.

The person recognition unit 120 acquires a facial image of a customergenerated by the facial recognition camera 220 and recognizes a personcontained in the acquired facial image. The person recognition unit 120refers to the suspicious person information DB 170 and makes comparisonof it with the facial image, and thereby determines whether the customeris a suspicious person or not. The in-store situation analysis unit 130acquires an in-store image generated by the in-store camera 230,analyzes the number of customers in the store based on the acquiredin-store image and detects the congestion in the store.

The alert information generation unit 140 generates alert information tobe transmitted to a surveillant based on detection results of thedistance image analysis unit 110, the person recognition unit 120 andthe in-store situation analysis unit 130, and outputs the generatedalert information to the alert device 240. The alert informationgeneration unit 140 generates and outputs the alert information based onthe hand action of a customer or a store staff detected by the distanceimage analysis unit 110, the alert information based on the suspiciousperson recognized by the person recognition unit 120, and the alertinformation based on the congestion in the store analyzed by thein-store situation analysis unit 130. Further, the alert informationgeneration unit 140 may record the generated alert information in a 3Dvideo of the 3D video information recording unit 160.

The suspicious action information DB 150 stores suspicious actionpatterns (suspicious action pattern information) for detecting asuspicious action of a customer or a store staff. Note that a suspiciousaction is an action (preliminary act) that raises suspicion ofmisconduct by a person such as a customer or a store staff. Thesuspicious action information DB 150 stores a product fraudulentacquisition pattern 151, a product fraudulent change pattern 152, amoney fraudulent acquisition pattern 153 and the like, for example, asthe suspicious action patterns.

The product fraudulent acquisition pattern 151 is pattern informationabout actions of fraudulently acquiring a product, which include, forexample, an action of a customer that puts a product in an improperplace other than a shopping basket or cart. The product fraudulentchange pattern 152 is pattern information about actions of fraudulentlychanging a product, which include, for example, an action of a customerthat breaks or damages a product. The money fraudulent acquisitionpattern 153 is pattern information about actions of fraudulentlyacquiring money, which include, for example, an action of a store staffthat puts money from a cash register in an improper place such as apocket of the store staff.

The suspicious person information DB 170 stores suspicious personidentification information for detecting that a customer who comes to astore is a suspicious person. The suspicious person includes a personwith a previous record, a habitual offender and a person on a blacklist,and the suspicious person identification information contains the name,gender, age, facial image information (image) and the like. For example,the suspicious person information DB 170 acquires and stores suspiciousperson information such as persons with previous records from a cloud(cloud network) 250 or the like, and further stores suspicious personinformation such as habitual offenders (persons on the blacklist) basedon the history in the store.

FIG. 4 shows the configuration of the distance image analysis unit 110in the security device 100. As shown in FIG. 4, the distance imageanalysis unit 110 includes a distance image acquisition unit 111, aregion detection unit 112, a hand tracking unit 113, and a hand actionrecognition unit 114. Although the elements for recognizing a person'shand action are mainly described below, a person's face, line of sight,product, money and the like can be detected by elements similar to thosefor recognizing a person's hand action.

The distance image acquisition unit 111 acquires a distance imagecontaining a customer or a store staff which is taken and generated bythe 3D camera 210. The region detection unit 112 detects a region ofeach part of a customer or a store staff contained in the distance imageacquired by the distance image acquisition unit 111.

The hand tracking unit 113 tracks the action of a hand of a customer ora store staff detected by the region detection unit 112. The hand actionrecognition unit 114 recognizes a suspicious action of the customer orthe store staff based on the hand action tracked by the hand trackingunit 113. For example, based on the suspicious action information DB150, the hand action recognition unit 114 determines whether thesuspicious action corresponds to a product fraudulent acquisitionpattern such as putting a product in a pocket, a product fraudulentchange pattern such as breaking a product, or a money fraudulentacquisition pattern such as putting money in a pocket of clothes.

A security method (security process) that is performed in the securitysystem (security device) according to this exemplary embodiment isdescribed hereinafter with reference to FIG. 5.

As shown in FIG. 5, a customer enters a store and comes close to a shelfin the store (S101). Then, the facial recognition camera 220 in thestore generates a facial image of the customer, and the security device100 checks the facial image against suspicious person information suchas a list of persons with previous records/on the blacklist (S102).Specifically, the person recognition unit 120 of the security device 100compares the facial image taken by the facial recognition camera 220with facial image information of suspicious persons (a list of personswith previous records/on the blacklist) stored in the suspicious personinformation DB 170 and searches for a person regarding which the facialimage and facial image information match and thereby determines whetherthe customer is a suspicious person or not.

After that, the customer performs a suspicious action such as putting aproduct in a place other than a shopping basket or cart (S103). Then,the 3D camera 210 in the vicinity of the shelf takes an image of thecustomer's hand, and the security device 100 recognizes the action ofthe customer's hand by using the distance image of the 3D camera 210(S104). Specifically, the distance image analysis unit 110 in thesecurity device 100 tracks the distance image of an image of thecustomer's hand, and recognizes that the customer has picked up theproduct and put it in an improper place.

Then, the security device 100 determines that a suspicious action hastaken place based on the customer's hand action recognized in S104, anddisplays and records an alert on the alert device 240 such as a storestaff terminal or a security guard terminal (S105). Specifically, thealert information generation unit 140 of the security device 100generates and outputs alert information indicating a determination thata suspicious action has taken place. Further, the alert informationgeneration unit 140 generates and outputs alert information based on thesuspicious person recognized in S102.

Further, besides a customer, a store staff comes close to a checkoutstand (S106) and performs a suspicious action such as putting money in aplace other than a cash register (S107). Then, in the same manner as inthe case of the customer's action, the security device 100 recognizesthe action of the store staff's hand by using the distance image of the3D camera 210 (S104), and displays and records an alert on the alertdevice 240 (S105).

FIG. 6 shows the details of a recognition processing (trackingprocessing) performed by the distance image analysis unit 110 in S104 ofFIG. 5. Note that, the processing shown in FIG. 6 is one example, andthe action of a hand may be recognized by another image analysisprocessing, and a person's face or line of sight, a product, money andthe like may be detected in the same way.

As shown in FIG. 6, the distance image acquisition unit 111 firstacquires a distance image containing a customer or a store staff fromthe 3D camera 210 (S201). Next, the region detection unit 112 detects aperson who is a customer or a store staff contained in the distanceimage acquired in S201 (S202) and further detects each region of theperson (S203). For example, the region detection unit 112 detects aperson (customer or store staff) based on the image and the distancecontained in the distance image by using a discrimination circuit suchas SVM (Support Vector Machine), and estimates the joint of the detectedperson and thereby detects the bone structure of the person. The regiondetection unit 112 detects the region of each part such as the person'shand based on the detected bone structure.

Then, the hand tracking unit 113 tracks the hand action of the customeror the store staff detected in S203 (S204). The hand tracking unit 113tracks the bone structure of the customer's hand and its vicinity anddetects the action of the fingers or palm of the hand based on the imageand the distance contained in the distance image.

After that, the hand action recognition unit 114 extracts the feature ofthe action of the hand based on the action of the hand tracked in S204(S205), and recognizes a suspicious action of the customer or the storestaff based on the extracted feature (S206). The hand action recognitionunit 114 extracts the direction, angle, and change in movement of thefingers or the palm (wrist) as a feature amount.

For example, the hand action recognition unit 114 detects that thecustomer is holding the product from the angle of the fingers, and whenthe customer moves the fingers off the product with the hand being closeto a pocket of clothes, it detects that the customer puts the product inthe pocket of the clothes. Then, the hand action recognition unit 114compares the detected action pattern with the product fraudulentacquisition pattern 151, the product fraudulent change pattern 152 andthe money fraudulent acquisition pattern 153, and when the detectedpattern matches any of those, it determines that it is a suspiciousaction. Further, the features of images of the product fraudulentacquisition pattern 151, the product fraudulent change pattern 152 andthe money fraudulent acquisition pattern 153 may be learned in advance,and the state of the hand may be identified by comparing a detectedfeature amount with the learned feature amount.

FIG. 7 shows the details of an alert output processing performed in S104and S105 of FIG. 5.

As shown in FIG. 7, the distance image analysis unit 110 determineswhether a customer or a store staff performs a suspicious action (S301).For example, the distance image analysis unit 110 determines an actionof a customer that puts a product in a place other than a shoppingbasket or cart, such as a pocket or a bag in hand, or an action thatbreaks a product, damages the product, adds a foreign body, orfraudulently changes the placement. Further, the distance image analysisunit 110 determines an action of a store staff that embezzles the sales,appropriates a product (puts money in a pocket etc), or gives a productto a customer as a conspirator without receiving a full price. Besides,the distance image analysis unit 110 determines, as a suspicious action,an action such as frequently looking around, tampering or illicitlymanipulating a pachinko machine, attaching a skimming device to an ATM(Automated Teller Machine), illicitly manipulating a cash register or anelectronic money device, or paying public money or issuing a ticketwithout through a regular process. When a customer or a store staffperforms the corresponding action, the following process is performed tooutput alert information in accordance with that action.

Specifically, the alert information generation unit 140 acquiressuspicious person information (S302) and acquires the congestion in astore (S303). The alert information generation unit 140 acquires thesuspicious person information indicating whether the customer is asuspicious person or not from the person recognition unit 120 so as toprovide the alert information on the basis of a person recognitionresult by the facial image of the facial recognition camera 220.Further, the alert information generation unit 140 acquires thecongestion in the store from the in-store situation analysis unit 130 soas to provide the alert information on the basis of a congestionanalysis result by the in-store image of the in-store camera 230. Notethat, before recognizing a customer, before analyzing the congestion,and when each information is not necessary, the acquisition of thesuspicious person information and the acquisition of the congestion maybe omitted.

Then, the alert information generation unit 140 determines a suspicionlevel of the detected suspicious action of a customer or a store staff(S304). For example, the suspicion level is assigned to each suspiciousaction pattern in the suspicious action information DB 150, and thesuspicion level is determined by referring to the suspicious actioninformation DB 150.

FIG. 8 shows examples of the suspicion level of a suspicious action. Forexample, as shown in FIG. 8, a level is set among the suspicion levels 1to 5. As the suspicion level is higher, it is more likely to bemisconduct, for example.

As one example, when a customer's action corresponds to the productfraudulent acquisition pattern “put a product in an improper place”, itis set to the suspicion level 3 by referring to the suspicious actioninformation DB 150. Then, the suspicion level is adjusted inconsideration of other parameters.

For example, when the time period of the suspicious action is short,when the quantity of products is large, or when the product isexpensive, it is likely to be shoplifting. Thus, thresholds are set forthe action time, the quantity of products, and the price of a product.When the time is shorter, when the quantity of products is larger, orwhen the price of a product is higher than such threshold, the suspicionlevel is set higher in accordance with the amount exceeding thethresholds. On the other hand, when the time is longer, when thequantity of products is smaller, or when the price of a product is lowerthan such threshold, the suspicion level is set lower in accordance withthe amount falling below the thresholds.

Likewise, when a customer's action corresponds to the product fraudulentchange pattern “damage a product”, “open a product box”, or “deform aproduct box”, it is set to the suspicion level 3 by referring to thesuspicious action information DB 150. Then, the suspicion level isadjusted in consideration of other parameters.

For example, when the damage is large or when the product box isdeformed, when the time period of the suspicious action is short, whenthe quantity of products is large, or when the product is expensive, itis likely to be malicious. Thus, thresholds are set for the scale (rate)of the damage, the action time, the quantity of products, and the priceof a product. When the damage is larger, when the time is shorter, whenthe quantity of products is larger, or when the price of a product ishigher than such threshold, the suspicion level is set higher inaccordance with the amount exceeding the thresholds. On the other hand,when the damage is smaller, when the time is longer, when the quantityof products is smaller, or when the price of a product is lower thansuch threshold, the suspicion level is set lower in accordance with theamount falling below the thresholds.

Likewise, when a store staff's action corresponds to the moneyfraudulent acquisition pattern “move money to an improper place”, it isset to the suspicion level 3 by referring to the suspicious actioninformation DB 150. Then, the suspicion level is adjusted inconsideration of other parameters.

For example, when the time period of the suspicious action is short, orwhen the amount of money is large, it is likely to be misappropriation.Thus, thresholds are set for the action time and the amount of money.When the time is shorter or when the amount of money is larger than suchthreshold, the suspicion level is set higher in accordance with theamount exceeding the thresholds. On the other hand, when the time islonger or when the amount of money is smaller than such threshold, thesuspicion level is set lower in accordance with the amount falling belowthe thresholds.

Further, when the customer is a suspicious person or when the situationin the store is congested/sparsely populated, the possibility of asuspicious action increases, and the suspicion level is set higher. Inaddition, the suspicion level may be set higher for an action of acustomer or a store staff that looks around.

Note that, although the suspicion level is determined based on thepresence or absence of a suspicious person or the state in a store aftera suspicious action is detected in this example, it may be determinedbefore a suspicious action is detected. For example, when a customer isa suspicious action, when the state in a store is congested/sparselypopulated, or when a customer has an open-top bag, alert information(warning information) may be output even when a suspicious action is notyet detected. Further, in such a case, the thresholds for detection of asuspicious action (such as the time to detect a suspicious action) maybe lowered to make a suspicious action easy to be detected.

After that, the alert information generation unit 140 outputs alertinformation based on a determination result (S305). The alertinformation generation unit 140 outputs the detected suspicious actionand the determined suspicion level to the alert device 240. For example,the output of the alert information may be controlled based on thesuspicion level. When the suspicion level is lower than a specifiedlevel, an alert is not necessary and the alert information needs not tobe output to the alert device 240. Further, in this case, the alertinformation may be only recorded in the alert device 240 without beingdisplayed thereon.

Further, the suspicious actions and the suspicion levels may be recordedin 3D video information. By recording the suspicion levels, it ispossible to extract and check only a part with a high suspicion leveland thus efficiently check the 3D video.

As described above, in this exemplary embodiment, the hand motion of acustomer or a store staff is observed by the 3D camera placed at thepotion from which a product shelf and a client (shopper) in front of theshelf can view to recognize a suspicious action of the customer or thestore staff. When a suspicious action is recognized, alert information(alarm) is notified to an employee terminal in a shelf, a cash register,a guard's room or a store, and the action is recorded,

Because it is thereby possible to precisely grasp the hand motion by the3D camera and thereby grasp the action such as fraudulently acquiring aproduct, fraudulently changing a product or fraudulently acquiringmoney, it is possible to accurately detect a suspicious action andoutput alert information in accordance with the suspicious action. It isthereby possible to automate the surveillance of a suspiciousaction/misconduct and efficiently enhance the security, therebyimproving the profit ratio.

Second Exemplary Embodiment

A second exemplary embodiment is described hereinafter with reference tothe drawings. This exemplary embodiment is an example that, as acomplement to the suspicious action recognition in the first exemplaryembodiment, detects a suspicious action by detecting a deviant actionfrom normal action information patterns. Specifically, the presentinvention is not limited to the example of the first exemplaryembodiment, and a suspicious action may be determined by detecting adeviation from a store staff's normal action such as work at a cashregister, rather than detecting an action of a store staff thatembezzles the sales, appropriates a product (puts money in a pocketetc.) or gives a product to a customer as a conspirator withoutreceiving a full price and the like.

FIG. 9 shows a configuration of a security system according to thisexemplary embodiment. As shown in FIG. 9, in this exemplary embodiment,the security device 100 further includes a normal action information DB180 in addition to the elements of the first exemplary embodiment shownin FIG. 2.

The normal action information DB 180 stores normal action patterns(normal action pattern information) indicating normal actions of acustomer and a store staff. The normal action information DB 180 storesa product normal acquisition pattern 181, a product normal changepattern 182, a money normal acquisition pattern 183 and the like, forexample, as the normal action patterns.

The product normal acquisition pattern 181 is pattern information aboutactions of normally acquiring a product, which includes an action of acustomer that puts a product in a shopping basket or cart, for example.The product normal change pattern 182 is pattern information aboutactions of normally altering a product, which includes an action of astore staff that changes the display of a product, for example. Themoney normal acquisition pattern 183 is pattern information aboutactions of normally acquiring money, which includes an action of a storestaff that performs normal cash register work and an action of a storestaff that gives money from a cash register to a customer, for example.

FIG. 10 shows the configuration of the distance image analysis unit 110in the security device 100 according to this exemplary embodiment. Asshown in FIG. 10, a deviant action detection unit 115 is furtherincluded in addition to the elements of the first exemplary embodimentshown in FIG. 4.

The deviant action detection unit 115 detects whether an action of acustomer or a store staff is deviated from a normal action (suspiciousaction) or not based on the the hand action tracked by the hand trackingunit 113. The deviant action detection unit 115 refers to the normalaction information DB 180 and compares the detected action of a customeror a store staff with the product normal acquisition pattern 181, theproduct normal change pattern 182 and the money normal acquisitionpattern 183, and when the action of a customer or a store staff does notmatch any of them, determines that it is a suspicious action. Forexample, a suspicion level may be set in accordance with the degree ofdeviation from normal action patterns (the degree of mismatch).

Further, a suspicious action may be determined in consideration of bothof a detection result by the hand action recognition unit 114 using thesuspicious action patterns and a detection result by the deviant actiondetection unit 115 using the normal action patterns. For example, anaction may be determined as a suspicious action when any one of the handaction recognition unit 114 and the deviant action detection unit 115determines that it is suspicious, or an action may be determined as asuspicious action when both of the hand action recognition unit 114 andthe deviant action detection unit 115 determine that it is suspicious.

As described above, by detecting a suspicious action based on whether anaction of a customer or a store staff is deviated from the normal actionpatterns, not limited to detecting a suspicious action using thesuspicious action patterns in the first exemplary embodiment, it ispossible to detect a suspicious action more accurately.

It should be noted that the present invention is not limited to theabove-described exemplary embodiment and may be varied in many wayswithin the scope of the present invention.

While the invention has been particularly shown and described withreference to exemplary embodiments thereof, the invention is not limitedto these embodiments. It will be understood by those of ordinary skillin the art that various changes in form and details may be made thereinwithout departing from the spirit and scope of the present invention asdefined by the claims.

This application is based upon and claims the benefit of priority fromJapanese patent application No. 2013-185130, filed on Sep. 6, 2013, thedisclosure of which is incorporated herein in its entirety by reference.

REFERENCE SIGNS LIST

-   1, 10 SECURITY SYSTEM-   11 IMAGE INFORMATION ACQUISITION UNIT-   12 TRACKING UNIT-   13 SUSPICIOUS ACTION DETECTION UNIT-   100 SECURITY DEVICE-   110 DISTANCE IMAGE ANALYSIS UNIT-   111 DISTANCE IMAGE ACQUISITION UNIT-   112 REGION DETECTION UNIT-   113 HAND TRACKING UNIT-   114 HAND ACTION RECOGNITION UNIT-   115 DEVIANT ACTION DETECTION UNIT-   120 PERSON RECOGNITION UNIT-   130 IN-STORE SITUATION ANALYSIS UNIT-   140 ALERT INFORMATION GENERATION UNIT-   150 SUSPICIOUS ACTION INFORMATION DB-   151 PRODUCT FRAUDULENT ACQUISITION PATTERN-   152 PRODUCT FRAUDULENT CHANGE PATTERN-   153 MONEY FRAUDULENT ACQUISITION PATTERN-   160 VIDEO INFORMATION RECORDING UNIT-   170 SUSPICIOUS PERSON DB-   180 NORMAL ACTION INFORMATION DB-   181 PRODUCT NORMAL ACQUISITION PATTERN-   182 PRODUCT NORMAL CHANGE PATTERN-   183 MONEY NORMAL ACQUISITION PATTERN-   210 3D CAMERA-   220 FACIAL RECOGNITION CAMERA-   230 IN-STORE CAMERA-   240 ALERT DEVICE-   300 PRODUCT SHELF-   301 PRODUCT-   302 MONEY-   310 CHECKOUT STAND-   311 CASH REGISTER-   400 CUSTOMER-   410 STORE STAFF

1. A security system, comprising: a memory storing instructions; and oneor more processors coupled to the memory, wherein the one or moreprocessors are configured to execute the instructions to: acquire inputimage information of a product shelf and a person in front of theproduct shelf taken from above the product shelf; detects a bonestructure of the person based on the input image information; track thebone structure of the person; detect an action of a hand of the personbased on the bone structure; determine a suspicious action based on theaction of the hand; and output alert information based on the determinedsuspicious action.
 2. The security system according to claim 1, whereinthe one or more processors are further configured to execute theinstructions to track an action of putting the product in a place otherthan a predetermined place, and wherein the predetermined place is ashopping basket or a cart.
 3. The security system according to claim 2,wherein the one or more processors are further configured to execute theinstructions to track the action of putting the product in a pocket ofclothes worn by the person.
 4. The security system according to claim 1,wherein the one or more processors are further configured to execute theinstructions to: record the suspicious action; and output the recordedsuspicious action.
 5. The security system according to claim 1, whereinthe one or more processors are further configured to execute theinstructions to: set a level of the suspicious action according tocontent of the suspicious action; and output the suspicious actioncorresponding to a new level higher than a predetermined level.
 6. Asecurity method comprising: acquiring input image information of aproduct shelf and a person in front of the product shelf taken fromabove the product shelf; detecting a bone structure of the person basedon the input image information; tracking the bone structure of theperson; detecting an action of a hand of the person based on the bonestructure; determining a suspicious action based on the action of thehand; and outputting alert information based on the determinedsuspicious action.
 7. The security method according to claim 6, whereintracking the bone structure of the person comprises tracking an actionof putting the product in a place other than a predetermined place,wherein the predetermined place is a shopping basket or a cart.
 8. Thesecurity method according to claim 6, wherein tracking the bonestructure of the person comprises tracking an action of putting money ina pocket of clothes worn by the person.
 9. The security method accordingto claim 6, further comprising: recording the suspicious action; andoutputting the recorded suspicious action.
 10. The security methodaccording to claim 6, further comprising: setting a level of thesuspicious action according to content of the suspicious action; andoutputting the suspicious action corresponding to a new level higherthan a predetermined level.
 11. A non-transitory computer readablemedium storing a security program causing a computer to perform asecurity method, the method comprising: acquiring input imageinformation of a product shelf and a person in front of the productshelf taken from above the product shelf; detecting a bone structure ofthe person based on the input image information; tracking the bonestructure of the person; detecting an action of a hand of the personbased on the bone structure; determining a suspicious action based onthe action of the hand; and outputting alert information based on thedetermined suspicious action.
 12. The non-transitory computer readablemedium according to claim 11, wherein tracking the bone structure of theperson comprises tracking an action of putting the product in a placeother than a predetermined place, wherein the predetermined is ashopping basket or a cart.
 13. The non-transitory computer readablemedium according to claim 11, wherein tracking the bone structure of theperson comprises tracking an action of putting money in a pocket ofclothes worn by the person.
 14. The non-transitory computer readablemedium according to claim 11, wherein the security program causes thecomputer to further perform: recording the suspicious action; andoutputting the recorded suspicious action.
 15. The non-transitorycomputer readable medium according to claim 11, wherein the securityprogram causes the computer to further perform: setting a level of thesuspicious action according to content of the suspicious action; andoutputting the suspicious action corresponding to a new level higherthan a predetermined level.
 16. The security system according to claim1, wherein the one or more processors are further configured to executethe instructions to output a warning information based on a state in astore before determining the suspicious action.
 17. The security methodaccording to claim 6, wherein outputting a warning information based ona state in a store occurs before determining the suspicious action. 18.The non-transitory computer readable medium according to claim 11,wherein the security program causes the computer to further performoutputting a warning information based on a state in a store beforedetermining the suspicious action.